There was just a meierhof and tavern in the places where the today´s village Koloděje, formerly Kaladý, is now. After the takeover of the property by Adam Čabelický of Soutice in 1565, the owner built for his comfort a moderate aristocratic house in the renaissance style with a trapezoid base which was fortified with a moat watered by the Lužnice river.

The owners of the manor changed very often. They were Jan Mencelius of Kolsdorf (born 1625), Jan de Vitte of Liliental, Johanka Branštejnová of Malovice, Pavel Michna of Vacinov. In 1704 the homestead was put up for auction and got into the ownership of the count František Wratislav of Mitrowitz, and with its family it was bound for centuries. The description of the fortress survived in the record of the Domesday Book from that year. The fortress was built of stone and covered with shingles; there was a mill with two unloadings, arched kitchen, one room, basement and pantry on the lowered ground floor. There were five rooms on the first floor, three rooms and a spacious dining room on the second floor.
In 1737-1741František Karel Wratislav and his wife Marie Anna née Kinská made the expansion of the residential part with the extension of the chapel which gave the castle its baroque look.
The Chapel of Saint Anna is a piece of work of the constructor Jan Hybner who was strongly influenced by the works of a baroque architecture master Kilián Ignác Dienzenhofer (1689-1751). The interior is richly decorated with frescoes, the biggest one is on the ceiling and it portrays the Virgin Mary with the Holy Trinity. On the walls the fresco turns into the illusive architecture, it displays handrails, columns etc. It is said the central altar picture of Saint Anna was made by Petr Brandl.
The wall paintings on the first floor come also from the end of the 18th century. They portray scenes from the life of one the ancestors of the Wratislavs of Mitrowitz, Václav called “Tureček” (1576-1635) who, as a fifteen-year-old page, took part in the journey of the bearers of the Emperor Rudolf II to Constantinople. He recorded his memories of the captivity, which followed, in his own book of travels “Příhody Václava Vratislava, svobodného pána z Mitrovic” (The stories of Václav Vratislav, a free master of Mitrovic).

The later modification happened in 1840, when the mill which was in the original renaissance part of the castle was shut down. The mill was relocated to a new building on the opposite shore of the Lužnice river and the whole riverbed was moved farther from the castle. Thus, the space in front of the castle enlarged and the park was enlarged as well, the park was originally established in 1753. In 1875 the family crypt of the dynasty of the Wratislavs of Mitrowicz was built between the castle and the entrance gate. The remains of the brothers Eugen and Karel Wratislav, carried from the charnel house of the graveyard from Týn nad Vltavou, were laid there.

The Dynasty of Wratislavs of Mitrowicz and Schoenfeld died without an heir with the death of Evžen Wratislav in 1943. His sister Marie Terezie (1884-1961) became the sole owner of the Koloděje Castle; she got married to a single master Dercsényi, the owner of a manor estate in Dolní Počernice. The property of the Dercényis was nationalized in the 1940s.

The local Collective farming administered the castle site together with the adjacent buildings of the brew house, distillery, stable and other objects. The castle building was forcibly separated with a tall wall and served as a museum depository. In the middle of 1960s the level of Lužnice river increased because of the construction of the Orlík dam, the weir disappeared and the adjacent park was reduced by the riverbed extension and building of protective anti-flood bank. In the beginning of the 1990s the heiress of the dynasty of Derscenyis (the dynasty died without an heir in 1987) made the claims of restitution and after acquiring the property they sold it in parts. Since 2008 the castle has been cited among the Protected Objects, it is under a total, very demanding reconstruction in the ownership of the company Mitrowicz, a.s. In 2016 the castle was opened to the public for more than a day for the first time in its almost five-hundred-year-long history and it offered a rich holiday programme to the visitors.