Drawing on the natural tones of Pasadena's famous Craftsman residences, the formal geometry of its Beaux Arts City Hall, and sparse elegance of later Streamline Moderne and Late Moderne commercial buildings, architects Thornton Ladd and John Kelsey designed the Norton Simon Museum to blend into and reference its urban surroundings.

Ladd & Kelsey were local Pasadena architects who met as students in USC's School of Architecture. They endeavored to create a museum that would blend between the residential area of Orange Grove Blvd and the commercial zone of Colorado Boulevard. The low-scale, H-shaped museum opened in 1969 as the Pasadena Art Museum.

The curved exterior walls of the museum are lined with 115,000 vertically stacked umber-colored tiles designed by ceramicist Edith Heath. The museum's original interior featured spacious curving galleries that facilitated the exhibition of large modern art pieces, which had been the focus of the institution since its founding in 1924. Norton Simon, a businessman and art collector, assumed control of the Pasadena Art Museum as it faced financial problems in the early 1970s.

Simon's collection was combined with the museum's existing collection, requiring some renovations to the galleries in the following decades. Both Craig Ellwood and Frank Gehry applied their personal touches the original design; Gehry's alterations in 1999 included the opening of sky lights to allow filtered natural light into the galleries.