So I noticed that tlb was just acting sluggish lately… couldn’t figure out what was up…. thought it might be the network. So I had some time after a sales call the other day to swing by the ISP. When I was there I was supposed to do a changeip on the server to a new IP block. I get there and the boot drive in the Xserve is solidly on – not a good sign. Also everything on the Xserve is going at about 3% of normal speed. All indicators point to HD going bad. At this point I make the executive decision to pull the drive and bring it back to the house to salvage what I can off it. Couldn’t have done it at a better time – it literally was going through the death throes as I worked on it. Hundreds of read errors, etc. I pull off most of the data but I’m unable to BOOT to it – somehow it locked itself into a read only mode and realized it was sick but allowed me to pull data.
I pull off the postgres data, mysql data and as much nntp/torrent stuff as I can… Then I realize I don’t need the torrent stuff to put it back online. So I begin the tedious work of trying to get everything back to normal.
The steps
1) Postgres data was not dumped but rather pulled from a directory. Problems that occurred after that – the version of postgres that I downloaded to the new HD was not compatible with the old data. OK – I’m a smart guy – I remember that I loaded the older postgres bin onto my G5 at home so I move the data there in anticipation of dumping from there. NO dice. Start getting out of memory kernal issues,. Criminey what is that about? After some more googling I finally found a page that gave me the settings I needed to BOOT up postgres, then do the dump.
Still to fix – it looks like somehow the avatar info got hosed along the way. The psql upload table is suspect.
2) OK so I get the psql data dumped, then bring it in, but nothing is working. I remember to set register_globals to ON, but still no dice. Finally realize that the php that comes with OS X Server is not compiled with –with postgres so I can’t use it. No prob – I go and hunt down the latest php from entropy.ch….. And think I have it made in the shade…..
BZZZZZZZZT
3) Postgres works fine at this point but for some damn reason I can’t log in to my mysql stuff through apache. I literally burn 3 nights researching this damn problem, all the while not realizing that the problems point to the entropy.ch php install. Finally Chunk points in that direction and I go researching tonight on it. Sure enough the version of php 5.2.5-5 did NOT allow ppc 32 bit mode to use mysql authentications. There were some workarounds that some offered but none worked for me easily. So I fire up the sick server HD and pull off the old php5 dir from it, upload, relink and voila! it works again. Only 20 or so hours of my life that I can’t reclaim. CRIPES.
4) CVS. CVS was giving us fits but I realized it was probably a permissions thing. Got permissions set for it again and all is cool again there.
5) CreateHomeDir. This was failing but mainly due to lack of server.flexgames.com being added to my DNS record. For reverse DNS lookups this was crucial. For some reason Leopard Server uses reverse to even create a homedir. Once I had the server.flexgames.com added to my dns entry and allowed it to propagate, all was good again.
Still to tackle – the avatar upload issue. Getting NNTP going again. Getting bi-directional syncing going again.
Most important thing to tackle. Getting the SQL dumps going automatically and syncing back to the house. Have I learned my lesson yet?