Category Archives: Work

Server issues – the end is in sight.

After 5 months of frustration, headaches, nightmares and stress, the BladeForums servers are looking like they are stable and tasks are nearly complete. We just need to get DMARC up and running and that will be the final checkbox – stability and speed are already showing great improvements.

I wish this had been a smooth process, but unfortunately it was much more drawn out than it needed to be largely because of issues that were out of my control.

The server switchover was precipitated by discovering issues with the nightly backup schema – MySQL table locking along with I/O transfer rates were resulting in nightly downtimes during peak traffic thanks to hosting provider misconfigurations. We’d specified backups begin at 0400 across the board, and they only configured that for the web server and had the db server set for midnight.

On top of that, vBulletin 4.x has been End Of Life’d by Internet Brands and further development will not occur as they focus on version 5. Unfortunately, v5 is not scalable for sites of our size, so we had to implement a platform migration strategy to ensure site longevity instead of remaining on legacy software. Step one of this was moving to more up-to-date virtual machines that were capable of running the latest packages and modern software.

So, back in May we had 3 new VMs spun up and anticipated a clean and orderly migration to the new nodes. We configured the new virtual machines with a dedicated www server and two database servers – a master and slave, where the slave would have the master’s db replicated to it. Backups would then be performed on the slave server which would completely eliminate any table locking issues resulting in downtime. We did test imports and even did trial runs of a possible successor forums platform to see whether migration would be feasible as an all in one effort.

Instead we experienced a multiple month long nightmare. Weeks were spent trying to troubleshoot what was going on, but it boiled down to the servers just…. ‘going away’ randomly every few seconds. We proceeded to import the forums data and what was noticeable during soft testing turned into complete site failures during live traffic, so multiple migration attempts (each spanning 8+ hours of downtime) failed. We had days of outages as a result, requiring reverting to the old server architecture.

Eventually it was determined that the VMs were, quite simply, built off bad templates. Only instead of completely blowing away the 3 bad VMs and generating 3 fresh ones, our hosting provider chose to only create a new web vm and wanted us to try and make it work. Needless to say, all 3 required replacement, costing me hours of technical support and thousands of dollars in time wasted, customer frustration, and more.

Once we had working VMs, we had a relatively short downtime with an easy migration that went off without any major issues. What should have taken 4 hours had dragged on for more than 4 months.

For the last few weeks we have been battling a persistent issue with Sphinx search – we could manually run the indexer and have new posts appear, but the automated reindexing and delta indexing was not occuring via cron.  Yesterday afternoon we began to get that resolved and it was completely fixed by 1AM this morning.  With that, our last obstacle for full operation is implementing DKIM / DMARC to get into compliance with email handling standards.

After that is complete, we will be ready to begin Phase II of transitioning away from vBulletin to another forum software platform, one that will offer even greater advantages and vastly benefit the membership in many ways.

For those members reading this, I appreciate your patience and support.


Shopsite Template switchover & lots of work to do

ShopSite recently released version 12sp1 of their shopping cart software, and with it comes a host of new features that I’ve been wanting to incorporate for quite a while.  For those curious about what new features are in this version, feel free to explore.

One of the hurdles I’m facing is that when I purchased the existing site templates back in 2009 many of the features I needed were not available at that time.   Various work arounds were custom coded into the “new” templates, and some streamlined operations.  Unfortunately when you custom code, you wind up with legacy bits and incompatibilities with future versions.

One of the bits of custom code was a piece of PHP that reformatted plain text into HTML tables; this greatly sped up production time as anyone could type in the product specifications and the software would pretty it up without having to copy/paste dreamweaver code and hoping someone didn’t format it incorrectly.

Shopsite v12sp1 introduced several new product tags, with PRODUCT.Specifications being the one that primarily interests me at the moment.  Since we currently use PRODUCT.Field3 for our existing specifications table plain text, this should be a matter of

  • Copy & rename template from appropriate Shopsite theme
  • Find relevant PHP code in legacy templates (done)
  • Locate PRODUCT.Specifications tag in new template
  • Port over PHP code to reflect new changes.


  • Update CSS code within legacy PHP to utilize shopsite color tags specified within back ends.  Tags identified:
    • row2, -> replace with STORE.TableShadowColor?
    • specsTable,
    • specsTableTitle,
    • specsLable
  • Download existing PRODUCT.Field3 data and reimport it to PRODUCT.Specifications field.