Basic Search

Basic Search

Basic Site Search

 

Whats Up Index

We speak
 

 Contact me for Custom work
or One on One Platform Instruction 

follow me on: 


« New WishList page :) | Free access to the Advanced Manual!!! »
Tuesday
Oct302012

The folks at SquareSpace are Troopers

Talk about being dedicated to the service that they provide,  They are carring fuel up 17 stories to keep the generators running.

Many thanks here.

Bruce (RR)

 They have pictures here also http://status.squarespace.com/

 

Squarespace Sandy Updates

  • Update [6:25pm ET]

    We have confirmed that the fuel pump is now working. We can now operate off of fuel indefinitely and there are continuous deliveries scheduled. Additional issues remain, the largest of which being that our generator will need to be taken offline at some point to clean the fuel filter. We can avoid that downtime if Peer1 can provide a street level generator. If this generator failure does occur, we expect the level of downtime to be on the order of an hour, instead of the multi-day outage we were facing at the onset of this crisis. We remain in a difficult situation, but the most difficult challenge we were presented with is now behind us. As of this writing, Squarespace has miraculously avoided all downtime related to Sandy, but we still have many challenges ahead with respect to getting back to anything normal.

  • Update [4:54pm ET]

    A pump has been successfully installed that is carrying fuel to the roof fuel header, and it has been running for an hour. We are awaiting confirmation that the header and our tank are properly connected. The building is going to attempt to power up downed generators for other floors in 30 minutes.

  • Update [2:43pm ET]

    We were able to temporarily resume carrying fuel for an hour, which means we have about 4-5 hours of fuel left on the roof.

    Datacenter engineers are on site attempting to finish a more permanent pump solution, which we should know more about within the hour.

  • Update [11:16am ET]

    Some bad news.

    We may not be able to deliver more fuel, as the building is shutdown mode. Last estimate is that we have about 3 hours left. Things change constantly, and we will keep you updated.

  • Update [9:03am ET]

    A short update to start the day…

    We have plenty of fuel on hand. The PEER1 generator is running strong and is in the process of being filled again. The bucket brigade will start by 10AM to keep us running for another 8-10 hours before the next brigade begins.

    A hopeful update from the building: This building has 2 basements and both are flooded. As of 7AM today, the first basement has been completely pumped out. A broken water main was discovered which was ADDING water to the basement as the pumps ran yesterday. By using 3x the number of pumps overnight, they were able to outpace the water main, clear the basement and turn off the running water main. We have no ETA on ConEd power for the building as they have another full basement to drain which will likely take another day. In parallel, the building is making progress on a separate solution to pump fuel to their 5000 gallon tank.

    Finally, PEER1 is waiting for fuel pumps from NJ which would replace the human bucket brigade. Unfortunately, their trucks are waiting for gas in NJ.

    Thanks again for all of your support.

  • Update [11:09pm ET]

    The midnight bucket brigade has begun (photo below).

    When tonight’s brigade is done, we estimate we have a conservative 10 hours of capacity, allowing our 8am shift to begin tomorrow with low stress and also a switch to 120 minute refill sessions every 8 hours, versus yesterday’s continuous huffing of fuel up the stairs. Yeah, we’re getting good at this.

    The building has had crews pumping water out of the basement all day. Unfortunately, we have no ETA on the repairing of the basement fuel delivery systems until the building’s crews get down there. However, both the building and PEER1 have been putting together separate contingency fuel pumping systems. We hope one of these will be online in the next 24-36 hours so that fuel refills will involve very few people and simply take place via a truck at street level.

    Our greatest risk at this point may be the generator itself. We’ve kept it fueled, and have a good system in place, but anything can happen. Fingers crossed. Have a good night, and again, thanks for all the great tweets and emails. They mean a lot!

Terrific progress this afternoon.

We have a few hundred gallons of spare fuel on the roof along with a full tank, which will take us well into the night. On top of teams from Peer1, Fog Creek, and Squarespace, we were able to hire additional help from Brooklyn and Queens. Special thanks to Mike Mazzei and his team at Peer1 for unprecedented dedication these past days.

We have a big crew teaming up tonight for a midnight bucket brigrade, and another in the morning. Traffic is terrible with gridlock everywhere and no public transport, so we are scheduling carefully.

We are cautiously optimistic that fuel will be running to the roof by end of day tomorrow or the day after. Water is being successfully pumped from the basement at a rate of one foot per hour. Fifteen feet of water remain.

We have increasing confidence that we may avoid downtime. Of course, there are no guarantees and we are working hour-by-hour.

  • Update [9:30am et]

    The fuel truck has arrived with eight barrels of deisel. The bucket brigade continues to keep the generator supplied. Another truck of fuel expected by noon.

     

  • Update [8:33AM ET]

    We arrived at 6AM. We have enough fuel in the tank to keep our services running into the late morning. We are expecting two fuel deliveries between now and then. We have 25 people stationed throughout the building to relay more fuel to the roof once it arrives. Assuming we are as successful today as we were yesterday in bringing fuel up, we’ll have power for the rest of the day, buying more time to continue pumping water from the basement. Unfortunately, that is progressing very slowly.

    We are working hand-in-hand with Peer 1 and Fog Creek on logistics today, and all three of our companies have teams here to help. Everyone thanks you for the tremendous support we have received via email and on Twitter. It has really helped keep spirits high.

    -JH

Update [1:02am ET]

We continue to manually provide fuel to the generator, a plan which appears to be working, but is likely not sustainable. The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts, joined with the building, can see us through. We have loaded enough fuel to carry to the morning, at which point we’ll provide another status update.

Update [7:39pm ET]

We have been running hour by hour by manually carrying fuel to our generators (17 floors) with support from the building.  As the night goes on, this is becoming a bit more difficult to sustain, as fuel trucks will appear more intermittently. For now, we remain online. Thank you all for your patience. 

As before, we do not have a sustainable solution to the energy problem, but we persevere in our efforts to get more fuel on site and get a pump connected.

  • Update [2:07pm ET]

    We have an understanding that when the fuel header on the roof stops refilling our tank, we will likely be in a window where shutdown will occur within 2 hours. That has not happened yet, which is good. We have fuel at the building, but cannot tap into the damaged building infrastructure yet.

  • Update from Anthony [12:47pm ET]

    Anthony here. Just wanted to share a personal update now that I have a moment.  

    At around 9:00 this morning upon hearing about the fuel situation at Peer1, I decided to head out and see if I could lend a hand. The streets of Manhattan near where I live (Soho – not in the evacuation zones) are in not-so-bad shape right now, but the damage left from the flooding in the evacuation zones is significant and real.

    I’m sitting in our datacenter NOC at 75 Broad St. Not that it’s been pointed out to me, but there are beds set up on the tiled floor here from the great team at Peer1 who stayed to monitor the situation overnight. These guys have incredible commitment to keeping everything running, and it’s great to see.

    Normally, power loss would not be a major problem for our datacenter – Peer1 stayed online during the major Manhattan power outage in 2003 that lasted for days, and we preemptively shifted to backup power around 4:00pm yesterday predicting that Con Edison would be shutting off power in evacuation zones.  Given the nature of the flooding, this situation escalated greatly, submerging our reserve fuel in the basement, shutting off the elevators, and damaging the pumps required to get this fuel to the generator on the 17th floor.

    My reasons for coming to the datacenter were twofold: one was simply to help and do whatever I could to help us (and the whole building) stay online. The other was to send our systems team the absolute final signal to perform a clean shutdown of our infrastructure should we be moments away from total power loss. Generally, clean shutdowns are preferable to abrupt halts, since code halting in a known state is better than code halting in an unknown state.

    We had an initial warning that 10:45am was going to be the clean shutdown time. To determine how much time we have remaining, engineers are taking readings at particular time intervals to attempt to determine how quickly we are depleting. The tank readings are behaving somewhat erratically, as there is another mechanism replenishing it from a separate fuel header. Some of our recent readings seem more optimistic, but it is impossible to predict how much fuel remains in this header at this time. As of this writing, we have at least 45 minutes left.

    Bridges to the island are open right now, and we currently have a fuel truck en route. We have approval from the building to manually carry fuel up and we have a number of our team on-site to offer manual help as needed.  

    Unfortunately, I do not have more information on a final resolution to this issue. You should still expect Squarespace to go offline at some point because of the hurricane’s aftermath, but we will do our best to keep that downtime to a minimum. Once we have a reliable stream of fuel to the building, it will go online independently of any other grid issues related to ConEdison and lower Manhattan in general.

    We’ll continue to keep you posted. Thank you for your patience.

    A

  • Update [11:46am ET]

    We have team members in our Network Operations Center working to keep things online for as long as possible. At the moment, we expect sites to be up for at least another hour. 

  • Update [11:04am ET]

    Peer1 is going to implement a controlled shutdown of NY Data Center at 10:45 ET. 

  • Downtime Expected Due to Sandy

    Everyone, I have some unfortunate news to share.

    Our primary data center, Peer1, in Lower Manhattan lost power yesterday at about 4:30PM local time. At that time, we smoothly made the transition to generator power and took comfort over the fact that we had enough fuel to last three to four days. (Peer1 stayed online during the last 3 major natural disasters in the area, including a blackout that lasted for days.)

    At 8:30PM yesterday, we received reports that the lobby in the data center’s building was beginning to take on water. By 10:30PM, as is sadly the case in most of Lower Manhattan, Peer1’s basement had experienced serious flooding. At 5AM, we learned our data center’s fuel pumps and fuel tanks were completely flooded and unable to deliver any more fuel. At 8AM, they reported that the generators would be able to run for a maximum of four more hours.

    Unfortunately, this means that Squarespace will be offline soon (our estimate being at 12PM today). Be assured that while this will impact our availability, there is no chance of data loss or any other permanent effects. We have simply run out of power, backup power, and cannot access our fuel in a flooded basement.

    Our teams have been working tirelessly on contingency plans. We are working to bring the Squarespace systems back online as soon as possible. As you have probably read, all bridges and tunnels into and out of Manhattan are closed and large portions of the city remain without power. We will do everything in our power to get Squarespace running as soon as possible, and we will remain online for as long as it is safe.

    Squarespace support will remain available 24/7 during this downtime. We will post updates to Twitter via @squarespacehelp and urge you to follow us there.

    Our hearts go out to the many people who have lost their lives in this terrible tragedy and also to those who continue to suffer through the consequences of this historic storm.

    Thank you for your understanding.

Reader Comments

There are no comments for this journal entry. To create a new comment, use the form below.

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>