The biggest e-mail provider in Poland is out of service.

Poland, 14:27 UTC:
The biggest Polish portal has some important services unavailable (like e-mail for a few million users...).

Evacuation or Flooded?
As of 20.00 UTC , May 18th, we are still receiving contradicting news. Some sources confirm that the server-room(s) of ITI have been flooded, but other sources confirm that the problem has been caused by evacuation of the equipment from the server-room of Onet.pl which is located in Cracow.

Parts of southern Poland, including Cracow, has been flooded due to heavy rain. Parts of Poland, but also the Czech Republic and Hungary have felt the full force of the weather.

Critical Infrastructure: 
After the critical situation with rains and flooding in Poland is over, it would be good for Polish authorities to investigate the preparedness of the Critical Infrastructure, defined as assets that are essential for the functioning of a society and economy, for such situation like we are facing now.

It's a good question, if the private company, today the biggest among on-line media in Poland, should be treated as part of the Critical Infrastructure or not?

Update: 

15:31 UTC: Situation with e-mail service provided by Onet is still not resolved. No official announcement from Onet.pl.

Update: 

16:21 UTC: Onet's e-mail service resumed.

(photo source:
first: bartosiewicz.pl
second: 24-7floodresponse.com)

May 12: DNS blackout in Germany

On May 12, 2010, Internet in Germany faced serious problems. According to German media (Focus, Spiegel, Chip, Frankfurter Allgemeine etc.) most of the web pages with addresses ending with .DE were not available between 11:30 UTC and 15:45 UTC. During the "blackout" DNS queries returned NXDOMAIN (Name Error RCODE), indicating probable existence of some errors during the process of uploading of the zone file into the DNS servers. This information has been briefly confirmed by Peter Koch from DENIC eG. It's quite likely that the defective or incomplete zone files have been uploaded into DNS servers.

Background

Usually domain registration and maintenance systems are separated from Domain Name System (DNS is resolving queries send by Internet users, translating domians into IP addresses). Normally there is no "direct" connection between registration system and live DNS. Those two "worlds" are connected when the registry system is exporting* zone file(s) and the zone file is uploaded into primary DNS server (and than distributed to the secondary Name Servers).

German accident is similar to the case from Spain (in 2006 the empty zone file has been uploaded) and Sweden (in 2009 incorrect DNSSEC-signed zone file has been uploaded). Unfortunately in both mentioned situations this critical part of the process has not been secured by automatic checks verifying the newly generated zone file.

Solution 

To avoid such incidents, automatic verification mechanism(s) can be used to check the differences between the new generated zone file and the previous one (the last correct zone file). If the number of changes is higher than usual, it indicates that some errors are quite likely to happen and the process should be terminated. It's the common practice in many different industry sectors, to check if the changes in the computer databases, physical storage, resources or energy consumption etc. are not deviated from what is expected from statistical estimations. The idea behind such statistical checks is to eliminate major errors using automatic verification systems. Of course such (statistical) methods don't eliminate minor problems, requiring more sophisticated control solutions to be implemented.

Explanations
*There is also another solution for updating zone files called "dynamic updates", allowing small portions of domains to up updated more frequently than zone files reloads. German registry is not using this method to update DNS.

12 maja - sądny dzień dla Internetu w Niemczech.

Internet w Niemczech w dniu 12 maja przeżył poważny problem. Jak donosiły niemieckie media, przez kilka godzin niedostępne były strony internetowe, których adresy domenowe były zakończone końcówką .DE. Podczas awarii, zapytania do DNS zwracały informacje, że domena nie istnieje czyli tzw. NXDOMAIN, co może wskazywać, że podczas ładowania plików stref na serwery DNS popełniono błąd i niepełne lub uszkodzone pliki stref znalazły się na serwerach DNS. Informację taką potwierdził Peter Koch z DENIC eG, podmiotu zarządzającego rejestracją i utrzymaniem domen z końcówką .DE, potwierdzając także, że awaria była odczuwalna pomiędzy 11.30 a 15.45 UTC, a więc przez ponad 4 godziny.

Rejestrując i używając domeny musimy pamiętać, że systemy rejestracji domen są niezależne z systemem DNS czyli serwerami odpowiadającymi na zapytania od użytkowników Internetu o adresy IP. Te dwa "światy" łączą się ze sobą w momencie, kiedy rejestr domen dokonuje tzw. eksportu plików stref. W przypadku Niemiec, taki eksport następuje co dwie godziny.

Analogiczny (identyczny) błąd jak rejestr z Niemiec, popełniły wcześniej rejestry z Hiszpanii (w 2006) oraz Szwecji (w 2009), gdzie umieszczono na serwerach DNS niepoprawne pliki stref. Niestety ten krytyczny etap, czyli przeniesienie informacji technicznych o delegowanych domenach z systemów rejestracji domen do systemu DNS, jak widać nadal jest problemem dla niektórych rejestrów domen w Europie.

Zarówno w przypadku Hiszpanii, Szwecji jak i Niemiec, nie dokonywano automatycznego sprawdzenia, czy wgrywany plik stref jest poprawny. Gdyby zastosowano mechanizmy weryfikacji, nie doszłoby do upublicznienia błędnych informacji.

Problem był bardzo szeroko komentowany w Niemczech i na całym świecie, nie tylko przez media specjalistyczne, ale także ogólno-informacyjne. W Polsce o problemie napisały:

• Webinside
• Internet Standard
• AZnews
• Dziennik Internautow