Skip to main content

pikka-bird-collector 0.1.0 Python ops monitoring tool released


Dear Pixelings,

pikka-bird-collector: Pikka Bird ops monitoring tool Collector component .

https://pypi.python.org/pypi/pikka-bird-collector
https://github.com/tiredpixel/pikka-bird-collector-py

I’m pleased to announce pikka-bird-collector 0.1.0—the very first release, providing support for gathering system load, CPU, memory, and disk metrics, and sending them to pikka-bird-server using both JSON and binary payloads.

Changelog:

  • first release! :D
  • pikka-bird-collector providing commands: collect
  • --eternal mode with automatic staggering
  • --format supporting JSON and binary payloads
  • system collector providing metrics: load, cpu, memory, disk

Peace,
tiredpixel

pikka-bird-server 0.1.0 Python ops monitoring tool released


Dear Pixelings,

pikka-bird-server: Pikka Bird ops monitoring tool Server component.

https://pypi.python.org/pypi/pikka-bird-server
https://github.com/tiredpixel/pikka-bird-server-py

I’m pleased to announce pikka-bird-server 0.1.0—the very first release, providing support for gathering metrics using both JSON and binary payloads from pikka-bird-collector.

Changelog:

  • first release! :D
  • pikka-bird-server providing commands: server, database-migrate
  • database schema principal tables: machines, services, collections, reports
  • GET / providing greeting
  • POST /collections for collecting reports, supporting JSON and binary payloads

Peace,
tiredpixel

sunrise

You’ve been walking all night. Yet another of those times, one more night within the uninterrupted line which stretches from all you’ve known to all you can imagine. Even now, it merges together into one long night, one long walk, one deep pain within your head somewhere behind the tears which have dried in the cold. No one knows where you are, not even you. Your phone’s been off for hours, although there’s no one you would call anyway. For all anyone else knows, you’re spending time with friends or doing whatever it is you do in between the weekly punctuation of how was your weekend it was okay thanks how was your weekend yeah really good saw some friends drank too much but yeah we had a good time.

The city’s quiet at night. After the evening with people going to restaurants dressed nicely, you pass to the late evening with people standing in lines waiting to be okayed by unsmiling bouncers. And you pass to the late late evening with people huddled in small groups behind barriers smoking, girls with bare legs looking really pretty cold. And you pass to the early night with occasional people drifting vaguely towards home, in ones not always successfully avoiding the patches of sick on the pavement, in twos arms around each other blind leading the blind. And then you pass to the night.

It’s properly cold by now, and the white lights stare brazenly and the yellow lights lurk suggestively. The streets are not really alive, every footfall is out of place, every new corner a trespass into a world which has far more to do with a graveyard than the bustling of shops and money under the sunlight. You’re not even trying to find your way, because to do that you’d have to know where you’re going. You walk turn right go straight turn right turn left go straight turn left, your subconscious flipping a coin for every decision. It doesn’t really matter.

For the first few miles, you have your earphones in, volume turned up way past the warning, playing the same song on repeat over and over and over again. Then, you abruptly shift a mental gear, stopping the music mid-track by yanking the headphones out of the socket. You change your phone from flight mode to off without looking, feeling the solitary buzz as you cut that connection. You continue to stare ahead fixedly, not focussing on anything, like the first few minutes after having used a laptop screen for twelve hours straight. Sleepwalking without sleeping. Steered by some strange magnetic fluctuation within the compass of the mind. The shoreline which has long given in.

You’ve been here before. Probably not this street, but a street very much like it. Maybe it was tonight, maybe it was a previous night. Maybe it’s the mere aftercolour of a dream some night when you fell asleep facedown on the floor after hours of crying into the carpet. Maybe it’s the echo of some other life. You walk turn left go straight turn left turn right go straight turn right. You try to think back to why this time, was anything really any different. You realise you’re looking at it all wrong, trying to reverse-apply some logic to something which wasn’t made using those tools. Because there wasn’t really anything different, nothing new. Like a groove being worn into a rock you know it was just more of the same. Yet another of those conversations, those days, those thoughts. You’re so tired of going round the same loop. You’re so bored of yourself for doing so. You know there’s nothing you can do about it.

Your breathing steadied a couple of miles back. You have no tears left to cry. There is nothing to understand, nothing to resolve, nothing to decide. Just more of the same. Yet another of those walks. You were hyperventilating a couple of times, the indescribable something welling up and choking you and stopping you from breathing whilst pounding your head relentlessly from inside. Flash after flash of lightning within your brain, electrifying your thoughts and freezing them in some unholy light which slows time and lingers over all the bitter regret for what you did and what was done to you, the immobilising fear about not just the future but the right here right now and how to make it to never mind next week but six seconds’ time. And all the while the banks of the river are flooding, and the darkness of the water seems thick and poisonous and you’re somewhere there in the middle of it miles away from anyone and miles away from anything and miles away from god and knowing that you’re drowning. You’re drowning, and nobody really understands.

After the night comes the early early morning. And with that, the sky colours a mere shade, then a little more, then a little more, the moon just hanging there in the growing light as if dead. Your pace slows, and you begin to notice your surroundings. Abandoned cars parked in rows layered with frost. Bins overflowing with dark sacks. Doors shut tight against the night. Bricks in the walls, slabs of paving, so much concrete. And the day keeps coming. Hidden birds launch into things they really need to say and have been lying awake all night thinking about. You hear the sporadic noise of cars on some road nearby, gaps of silence following each decrescendo.

And you realise you have survived the night. Here is another day, here is another time. The pain in your head has given way to the dull ache which will hang around for hours whilst it dissolves. You’re not really sure how you got through the last few hours. You know you can’t think about it, you know you wouldn’t understand. And you know that the night has left its mark, one more freezing and thawing which is taking its toll and leaving you emptier each time you go around the same loop. You’re not sure you’ve got one more of those nights in you.

Somewhere deep within your fractured soul, you know that you’ve lost something. You’re not sure exactly when or where, or even what it is. But it was important, really really important, and now it’s gone and you don’t know how to find it. You come across a train station, and it’s an hour until the first train. You sit down for the first time in hours. You turn your phone back on, switch off flight mode. No missed calls. No texts. It’s way too cold, and you almost regret having not brought a coat. Leaning against the outside wall of the station, you watch the sunrise.

New Scientist, Feedback, Year 2038 Problem

A letter I wrote to New Scientist in response to Feedback, p. 56, 31 Jan 2015, No. 3006, regarding their mention of the Year 2038 Problem:


Dear New Scientist,

Re: Feedback, p. 56, 31 Jan 2015, No. 3006

The ‘date in December 1901’ referenced is likely 1901-12-13 20:45:52 UTC [1]. However, this only applies to software programs storing dates as seconds calculated externally to the database, or programs using certain kinds of timestamp types [2]. But there is no need to do this, as major databases provide proper date handling which is more sophisticated than simply using 64 bits, and not vulnerable to the Year 2038 Problem. Using these main date types, PostgreSQL has a latest date in Year 294276 [3], and MySQL, Microsoft SQL Server, and Oracle Database in Year 9999 [4] [5] [6], assuming storage of the time component is required. However, there are also ways to mitigate against these limitations, too. Thus, [it] is likely that resolving the issue for embedded systems will indeed be more difficult.

Peace,
tiredpixel

[1]: Unix Epoch - 2^31 (c.f. the latest date, Unix Epoch + 2^31 - 1, because of 0)
[2]: MySQL ‘timestamp’ type, https://dev.mysql.com/doc/refman/5.7/en/datetime.html
[3]: PostgreSQL ‘timestamp’ type, http://www.postgresql.org/docs/9.4/static/datatype-datetime.html
[4]: MySQL ‘DATETIME’ type, https://dev.mysql.com/doc/refman/5.7/en/datetime.html
[5]: Microsoft SQL Server ‘datetime2’ type, https://msdn.microsoft.com/en-us/library/bb677335.aspx
[6]: Oracle Database ‘NUMBER’ type, http://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm

log shipping using Rsync over SSH

If you’re looking to ship logs from multiple servers and perform all sorts of magic, you might like to consider Logstash or Fluentd. But what if you want something really lightweight, or simply don’t have the time to configure a full-blown solution? Rsync over SSH can provide a simple alternative, provided that all you really want to do is have logs accessible from a central place and are a happy Grep user. You’ve probably got the packages installed already, too.

Aim: One or more servers shipping logs to a central log master, using Rsync over SSH. System: Ubuntu Server 14.04 LTS, managed using Puppet.

set up log satellites

Set up a new SSH key for the log satellites, which will be connecting to the log master. This outgoing user could be root, which has full read-access. If root concerns you and you choose another user, be sure it is a member of appropriate groups for read-access to /var/log/.

Distribute the SSH key, including the private component (which is why it must be a new key and not used for anything else), to the log satellites (e.g. /root/.ssh/id_rsa). This will be used to connect to the log master. Ensure that the log master is marked as a known host to prevent issues with the first connection; one way to do this is to use ssh-keyscan:

$logger = 'logger.example.com'

exec { "/usr/bin/ssh-keyscan ${logger} >> /root/.ssh/known_hosts":
  unless => "/bin/grep ${logger} /root/.ssh/known_hosts",
}

set up log master

Open Port 22 from the log satellites to the log master in your firewall. Create a new user (e.g. logmaster) on the log master for receiving the logs, authorising the public component of the SSH key you created, and choosing somewhere to store the shipped logs (e.g. /srv/logs). You’ll probably want to include this in your backups.

user { 'logmaster':
  password   => 'PASSWORD_HASH',
  managehome => true,
  shell      => '/bin/bash',
}->

ssh_authorized_key { 'logmaster/ops-root@example.com':
  type    => 'ssh-rsa',
  key     => 'PUBLIC_KEY',
  user    => 'logmaster',
  options => [
    'command="rsync --server -ltrze.iLs . /srv/logs/${SSH_CLIENT%% *}"',
    'no-agent-forwarding',
    'no-port-forwarding',
    'no-x11-forwarding',
  ],
}->

file { '/srv/logs':
  ensure  => directory,
  mode    => '0730',
  owner   => 'root',
  group   => 'logmaster',
}

The ssh_authorized_key options are to restrict the logmaster user to running only the command specified, with received logs stored in a subdirectory named after the log satellite IP address. This is harder than hostname for the log satellite to switch, although you might like to give thought to what you’d like to do if a log satellite reuses a previous IP address. (An easy alternative, although slower and more space-intensive, would be to copy /var/log to somewhere including the hostname before shipping, or somehow pass through the hostname. I’d love to hear if you’ve got a nice adjustment for this without relaxing the SSH command strictness.)

Here, the permissions on /srv/logs are to allow logmaster write-access without list-access on the top-level, restricting visibility from within the logmaster user.

schedule log shipping

All that remains is to schedule the logs to be shipped. The actual shipping should usually be pretty quick, perhaps only a second or two, so Cron is good for this. You’ll probably want to test the command manually, too. (If you have problems and need to debug, considering temporarily relaxing the logmaster user SSH command to allow you to test the connection more easily.)

cron { 'rsync_var_log':
  command => '/usr/bin/rsync -rltz /var/log/ logmaster@logger.example.com:',
  user    => 'root',
  hour    => '*',
  minute  => '*/5',
}

Here, no -delete is used, to work nicely with log rotation; if log satellites rotate log files to a timestamped filename and keep a maximum number, the log master should build up an archive without further ado. If you choose to alter the Rsync options, you’ll need to ensure that the logmaster user SSH command is changed to match; you can see which command is needed by connecting manually using the -e ‘ssh -v’ option and observing the remote Rsync command. No destination directory is needed, as this gets ignored anyway by the restrictions on the log master.

Peace,
tiredpixel ☮