The Old Town of Bilbao is known as Casco Viejo, or to the locals as "las 7 calles" due to the original 7 streets that formed the original medieval town. Today, it is still the heart of the city, and I highly recommend wandering through the neighborhood with its beautiful narrow streets, packed with colorful buildings. Whilst here, be sure to keep an eye out for the Cathedral de Santiago (St. James Cathedral) for its gothic architecture, or Plaza Nueva where you can find yourself some delicious pintxos in the surrounding bars and restaurants.
Sit outside on the terraces so you can enjoy some quality people watching time as well ;-) If you're looking to learn more about culture and history, seek out the Basque Museum.
