Skip to main content

2015 (old posts, page 1)

pikka-bird-collector 0.1.0 Python ops monitoring tool released


Dear Pixelings,

pikka-bird-collector: Pikka Bird ops monitoring tool Collector component .

https://pypi.python.org/pypi/pikka-bird-collector
https://github.com/tiredpixel/pikka-bird-collector-py

I’m pleased to announce pikka-bird-collector 0.1.0—the very first release, providing support for gathering system load, CPU, memory, and disk metrics, and sending them to pikka-bird-server using both JSON and binary payloads.

Changelog:

  • first release! :D
  • pikka-bird-collector providing commands: collect
  • --eternal mode with automatic staggering
  • --format supporting JSON and binary payloads
  • system collector providing metrics: load, cpu, memory, disk

Peace,
tiredpixel

pikka-bird-server 0.1.0 Python ops monitoring tool released


Dear Pixelings,

pikka-bird-server: Pikka Bird ops monitoring tool Server component.

https://pypi.python.org/pypi/pikka-bird-server
https://github.com/tiredpixel/pikka-bird-server-py

I’m pleased to announce pikka-bird-server 0.1.0—the very first release, providing support for gathering metrics using both JSON and binary payloads from pikka-bird-collector.

Changelog:

  • first release! :D
  • pikka-bird-server providing commands: server, database-migrate
  • database schema principal tables: machines, services, collections, reports
  • GET / providing greeting
  • POST /collections for collecting reports, supporting JSON and binary payloads

Peace,
tiredpixel

sunrise

You’ve been walking all night. Yet another of those times, one more night within the uninterrupted line which stretches from all you’ve known to all you can imagine. Even now, it merges together into one long night, one long walk, one deep pain within your head somewhere behind the tears which have dried in the cold. No one knows where you are, not even you. Your phone’s been off for hours, although there’s no one you would call anyway. For all anyone else knows, you’re spending time with friends or doing whatever it is you do in between the weekly punctuation of how was your weekend it was okay thanks how was your weekend yeah really good saw some friends drank too much but yeah we had a good time.

The city’s quiet at night. After the evening with people going to restaurants dressed nicely, you pass to the late evening with people standing in lines waiting to be okayed by unsmiling bouncers. And you pass to the late late evening with people huddled in small groups behind barriers smoking, girls with bare legs looking really pretty cold. And you pass to the early night with occasional people drifting vaguely towards home, in ones not always successfully avoiding the patches of sick on the pavement, in twos arms around each other blind leading the blind. And then you pass to the night.

It’s properly cold by now, and the white lights stare brazenly and the yellow lights lurk suggestively. The streets are not really alive, every footfall is out of place, every new corner a trespass into a world which has far more to do with a graveyard than the bustling of shops and money under the sunlight. You’re not even trying to find your way, because to do that you’d have to know where you’re going. You walk turn right go straight turn right turn left go straight turn left, your subconscious flipping a coin for every decision. It doesn’t really matter.

For the first few miles, you have your earphones in, volume turned up way past the warning, playing the same song on repeat over and over and over again. Then, you abruptly shift a mental gear, stopping the music mid-track by yanking the headphones out of the socket. You change your phone from flight mode to off without looking, feeling the solitary buzz as you cut that connection. You continue to stare ahead fixedly, not focussing on anything, like the first few minutes after having used a laptop screen for twelve hours straight. Sleepwalking without sleeping. Steered by some strange magnetic fluctuation within the compass of the mind. The shoreline which has long given in.

You’ve been here before. Probably not this street, but a street very much like it. Maybe it was tonight, maybe it was a previous night. Maybe it’s the mere aftercolour of a dream some night when you fell asleep facedown on the floor after hours of crying into the carpet. Maybe it’s the echo of some other life. You walk turn left go straight turn left turn right go straight turn right. You try to think back to why this time, was anything really any different. You realise you’re looking at it all wrong, trying to reverse-apply some logic to something which wasn’t made using those tools. Because there wasn’t really anything different, nothing new. Like a groove being worn into a rock you know it was just more of the same. Yet another of those conversations, those days, those thoughts. You’re so tired of going round the same loop. You’re so bored of yourself for doing so. You know there’s nothing you can do about it.

Your breathing steadied a couple of miles back. You have no tears left to cry. There is nothing to understand, nothing to resolve, nothing to decide. Just more of the same. Yet another of those walks. You were hyperventilating a couple of times, the indescribable something welling up and choking you and stopping you from breathing whilst pounding your head relentlessly from inside. Flash after flash of lightning within your brain, electrifying your thoughts and freezing them in some unholy light which slows time and lingers over all the bitter regret for what you did and what was done to you, the immobilising fear about not just the future but the right here right now and how to make it to never mind next week but six seconds’ time. And all the while the banks of the river are flooding, and the darkness of the water seems thick and poisonous and you’re somewhere there in the middle of it miles away from anyone and miles away from anything and miles away from god and knowing that you’re drowning. You’re drowning, and nobody really understands.

After the night comes the early early morning. And with that, the sky colours a mere shade, then a little more, then a little more, the moon just hanging there in the growing light as if dead. Your pace slows, and you begin to notice your surroundings. Abandoned cars parked in rows layered with frost. Bins overflowing with dark sacks. Doors shut tight against the night. Bricks in the walls, slabs of paving, so much concrete. And the day keeps coming. Hidden birds launch into things they really need to say and have been lying awake all night thinking about. You hear the sporadic noise of cars on some road nearby, gaps of silence following each decrescendo.

And you realise you have survived the night. Here is another day, here is another time. The pain in your head has given way to the dull ache which will hang around for hours whilst it dissolves. You’re not really sure how you got through the last few hours. You know you can’t think about it, you know you wouldn’t understand. And you know that the night has left its mark, one more freezing and thawing which is taking its toll and leaving you emptier each time you go around the same loop. You’re not sure you’ve got one more of those nights in you.

Somewhere deep within your fractured soul, you know that you’ve lost something. You’re not sure exactly when or where, or even what it is. But it was important, really really important, and now it’s gone and you don’t know how to find it. You come across a train station, and it’s an hour until the first train. You sit down for the first time in hours. You turn your phone back on, switch off flight mode. No missed calls. No texts. It’s way too cold, and you almost regret having not brought a coat. Leaning against the outside wall of the station, you watch the sunrise.

New Scientist, Feedback, Year 2038 Problem

A letter I wrote to New Scientist in response to Feedback, p. 56, 31 Jan 2015, No. 3006, regarding their mention of the Year 2038 Problem:


Dear New Scientist,

Re: Feedback, p. 56, 31 Jan 2015, No. 3006

The ‘date in December 1901’ referenced is likely 1901-12-13 20:45:52 UTC [1]. However, this only applies to software programs storing dates as seconds calculated externally to the database, or programs using certain kinds of timestamp types [2]. But there is no need to do this, as major databases provide proper date handling which is more sophisticated than simply using 64 bits, and not vulnerable to the Year 2038 Problem. Using these main date types, PostgreSQL has a latest date in Year 294276 [3], and MySQL, Microsoft SQL Server, and Oracle Database in Year 9999 [4] [5] [6], assuming storage of the time component is required. However, there are also ways to mitigate against these limitations, too. Thus, [it] is likely that resolving the issue for embedded systems will indeed be more difficult.

Peace,
tiredpixel

[1]: Unix Epoch - 2^31 (c.f. the latest date, Unix Epoch + 2^31 - 1, because of 0)
[2]: MySQL ‘timestamp’ type, https://dev.mysql.com/doc/refman/5.7/en/datetime.html
[3]: PostgreSQL ‘timestamp’ type, http://www.postgresql.org/docs/9.4/static/datatype-datetime.html
[4]: MySQL ‘DATETIME’ type, https://dev.mysql.com/doc/refman/5.7/en/datetime.html
[5]: Microsoft SQL Server ‘datetime2’ type, https://msdn.microsoft.com/en-us/library/bb677335.aspx
[6]: Oracle Database ‘NUMBER’ type, http://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm

log shipping using Rsync over SSH

If you’re looking to ship logs from multiple servers and perform all sorts of magic, you might like to consider Logstash or Fluentd. But what if you want something really lightweight, or simply don’t have the time to configure a full-blown solution? Rsync over SSH can provide a simple alternative, provided that all you really want to do is have logs accessible from a central place and are a happy Grep user. You’ve probably got the packages installed already, too.

Aim: One or more servers shipping logs to a central log master, using Rsync over SSH. System: Ubuntu Server 14.04 LTS, managed using Puppet.

set up log satellites

Set up a new SSH key for the log satellites, which will be connecting to the log master. This outgoing user could be root, which has full read-access. If root concerns you and you choose another user, be sure it is a member of appropriate groups for read-access to /var/log/.

Distribute the SSH key, including the private component (which is why it must be a new key and not used for anything else), to the log satellites (e.g. /root/.ssh/id_rsa). This will be used to connect to the log master. Ensure that the log master is marked as a known host to prevent issues with the first connection; one way to do this is to use ssh-keyscan:

$logger = 'logger.example.com'

exec { "/usr/bin/ssh-keyscan ${logger} >> /root/.ssh/known_hosts":
  unless => "/bin/grep ${logger} /root/.ssh/known_hosts",
}

set up log master

Open Port 22 from the log satellites to the log master in your firewall. Create a new user (e.g. logmaster) on the log master for receiving the logs, authorising the public component of the SSH key you created, and choosing somewhere to store the shipped logs (e.g. /srv/logs). You’ll probably want to include this in your backups.

user { 'logmaster':
  password   => 'PASSWORD_HASH',
  managehome => true,
  shell      => '/bin/bash',
}->

ssh_authorized_key { 'logmaster/ops-root@example.com':
  type    => 'ssh-rsa',
  key     => 'PUBLIC_KEY',
  user    => 'logmaster',
  options => [
    'command="rsync --server -ltrze.iLs . /srv/logs/${SSH_CLIENT%% *}"',
    'no-agent-forwarding',
    'no-port-forwarding',
    'no-x11-forwarding',
  ],
}->

file { '/srv/logs':
  ensure  => directory,
  mode    => '0730',
  owner   => 'root',
  group   => 'logmaster',
}

The ssh_authorized_key options are to restrict the logmaster user to running only the command specified, with received logs stored in a subdirectory named after the log satellite IP address. This is harder than hostname for the log satellite to switch, although you might like to give thought to what you’d like to do if a log satellite reuses a previous IP address. (An easy alternative, although slower and more space-intensive, would be to copy /var/log to somewhere including the hostname before shipping, or somehow pass through the hostname. I’d love to hear if you’ve got a nice adjustment for this without relaxing the SSH command strictness.)

Here, the permissions on /srv/logs are to allow logmaster write-access without list-access on the top-level, restricting visibility from within the logmaster user.

schedule log shipping

All that remains is to schedule the logs to be shipped. The actual shipping should usually be pretty quick, perhaps only a second or two, so Cron is good for this. You’ll probably want to test the command manually, too. (If you have problems and need to debug, considering temporarily relaxing the logmaster user SSH command to allow you to test the connection more easily.)

cron { 'rsync_var_log':
  command => '/usr/bin/rsync -rltz /var/log/ logmaster@logger.example.com:',
  user    => 'root',
  hour    => '*',
  minute  => '*/5',
}

Here, no -delete is used, to work nicely with log rotation; if log satellites rotate log files to a timestamped filename and keep a maximum number, the log master should build up an archive without further ado. If you choose to alter the Rsync options, you’ll need to ensure that the logmaster user SSH command is changed to match; you can see which command is needed by connecting manually using the -e ‘ssh -v’ option and observing the remote Rsync command. No destination directory is needed, as this gets ignored anyway by the restrictions on the log master.

Peace,
tiredpixel ☮

breakfast

She would never forget the moment she’d finally realised. It was a Saturday morning, a few months ago, sunny for once. They’d slept in, tired from the week before. He, with a job in the city, working late on some project, paying the bills, keeping the bosses happy, that sort of thing. She, still on maternity leave, working out how to care for this beautiful, screaming child, her first. Their first. He set the table for breakfast whilst she fetched milk, cereal, toast, black coffee. They always tried to make a bit more of an effort on weekends; a chance to slow the pace.

He’d not been himself, recently. She’d noticed gradually, like the leak in the floor above; not really noticeable at first, something not quite right, then water starting to seep through. She’d put it down to a bad week, then a bad month. They both seemed tired all the time, could never quite get the rest they needed. Things will improve when we’re not being woken every few hours, she thought. Just give him space. But the days turned into weeks, and she began to really worry. Mid-week, she’d been on the phone to a friend, confiding she didn’t know what was happening, what to do. To her surprise, her friend, although clearly concerned, sounded cautious—guarded, almost—as if there were something not being voiced. After putting down the phone, she stared at it in bemusement. And then it hit her. Oh god, he’s having an affair.

She didn’t get much sleep, that night. She lay next to him, staring up at the ceiling. The streetlamp directly outside the window leaked around the edges of the curtain, mixing the shadows with a pool of orange. She couldn’t cry, not at first, but as the hours drew on, the pain in the front of her head started to shift, and tears started to drip noiselessly down her face. Across her cheek, onto the pillow, each tear slightly wetter than the last. Like the ceiling. She fell into some sort of half-awake, half-asleep doze, somehow still aware of the streetlamp outside.

In the morning, she knew what she was going to do. She’d wait until he came home from work that evening, and then confront him. Tell him she knew. Demand that he explain himself. But she suddenly knew she couldn’t wait until evening. He came out of the shower, towel still wrapped around him, about to get dressed for work. He seemed a little surprised to see her in the hallway, leaning against a doorframe for support. Goodness knows what look she had on her face. With nothing like the self-assurance she’d been planning, she somehow found a voice. It didn’t sound like hers; it was smaller and much further away.

He was late for work, that day. At first he’d seemed shocked, then that quickly morphed into anger. That helped her confidence, and she realised that she was angry, too. They both shouted at the same time, standing there in the cramped hallway. The initial fuel burned out, and some listening mixed with the shouting. He swore blind there was no one else. He couldn’t believe she’d even thought it. Didn’t he always come home soon after work? When would he have the time for an affair, anyway? She’d never really known him to lie to her. She was glad to have her suspicions shot to shreds, not least because the more she believed him, the more the dark, drowning sense of guilt grew and grew and strangled her. Yes, she believed him. How could she have even thought such a thing? Where had that thought even come from? How had she even said that? She hated herself.

But all that had been a few days before. They hadn’t exactly said much to each other since, but she was hopeful things would smooth over. Some time together this weekend was what they needed. A chance to relearn how to trust each other. A chance for the immediate rawness in both their minds to fade. They were both just tired, after all. Perhaps work had been tougher than he’d been saying. She poured them each a coffee, and carefully picked up her child out of the cot. They took their place opposite him at the table.

Her child looked so beautiful, so trusting; it made her feel full of love. As she gazed, she was sure she saw what had to be two smiles, first at her, then at him. Her eyes brimmed with tears, but the happy kind. Things would work out; how could they not? She looked at him, expecting their eyes to meet as they shared that unspoken moment. But he hadn’t seen; he was reading something or other on his phone.

The second realisation was even worse than the first, and more permanent. As she sat there, locked in a toxic bubble of thoughts, she felt the life drain out of her. She suddenly couldn’t quite remember how to breathe. There seemed to be something stuck in her throat, strangling her from inside. The pain returned to the front of her head. She struggled against it, feeling her mind forcibly turning to look at some unspeakable horror she could see growing in her peripheral. The swirling of particles taking form. The electrified split-second before lightning strikes.

She’d somehow lost him.