I don't know much about what schools you've been looking at, and what exactly you want to focus on in architecture, but you could always get your bachelors in any unrelated field, or even interior architecture if you want, then apply for a masters in architecture when you're done with that. This was you'll have 2 different backgrounds rather than 1, and you won't have to go through the pain of switching schools. Interior architecture will also give you a feel of studio work/life and allow you to really see if it's what you want.

Firstly, I would agree with SalemWitch, you could most definitely get your bachelors in interior architecture at the school you're at now. I think you'll definitely get a great feel for "how to design". Then you could apply to the many M.Arch programs out there. From the sounds of it, how much you said you always loved buildings and architecture, that is pretty huge. Not many have that, in fact, I feel like it's a rare thing to have. Of course, I may just surround myself around too many people uninterested, but loving buildings and architecture is the beginnings of that passion needed to get through a program.

Secondly, I went to University of Nebraska-Lincoln. I received my Bachelors of Science in Design. It's basically a Bachelors degree in Architecture. It is not an accredited degree, you would have to get your M.Arch for that, which I also received. 4 years for BS in Design, additional 2 years for M.Arch. I do not know if another architecture program has that same format, but the BS in Design is Architecture based in the College of Architecture here in Nebraska. I can name 2 people I worked with that only got their BS in Design and were working at firms as capable architects. They couldn't get licensed, but their knowledge of the building industry was strong because they learned while practicing the profession, as most of us learn. 

In other words, yes, with my BS in Design, I learned about how to design buildings. It was not a general design degree, or art degree, it was an architecture degree, just not one you can use to become licensed. 

I am in no way endorsing the UNL College of Architecture program, I am just saying they have what you specifically asked for. (Nor am I saying it's a bad idea to go there. I had a great time while there, and it's a well respected program in the midwest.)

I would suggest this. Go to any architecture program. it could be the 4 year bachelors as I described, or it could be your typical 5 year B.Arch program anywhere else. Try your hand, learn and see if you like it. You will definitely know if you want to stick with it when your spending countless hours on model building, or hours on creating a visually effective boards that display your drawings, sections, and renderings. Good Luck!