How Jim Retired Early With a $3M Portfolio That Generates $120K Per Year... Tax Free

Earlier this year I was delighted when a reader I’ll call Jim replied to one of my newsletter emails.

(Aside: if any of you ever want to talk to me, feel free to reply to the emails. It really is that easy. I might not respond to every one, but I sure do read them all.)

After a few emails back and forth, it turns out Jim and I have a ton in common, and I found out he lived only a few hours away. I was heading through his neck of the woods one day, so we made plans for lunch. Jim took me to his favourite restaurant — Wendy’s naturally — and the next thing I knew we had spent three hours talking. Jim had really interesting things to say about building a dividend portfolio and financial independence.

I knew you guys would get a kick out of him, so I started harassing him for an interview. He quickly dismissed the idea, thinking that he didn’t have anything to say. According to Jim, his story wasn’t very interesting. I disagreed, so I decided to up the pressure a little bit.

Even Jim couldn’t say no to the many people who wanted to hear his story, so I prepared some questions. Jim’s responses were thoughtful, and perfectly illustrated why I so enjoyed spending time with him. He shared way more than I expected too, giving what was already going to be a valuable interview an extra oomph. I know I’m going to refer to Jim’s wisdom often. I hope you all get value from it, too.

Without further adieu, let’s get to it.

(Note that this interview has been slightly edited for length and clarity)

Jim, thanks so much for taking the time to do this. I think people are going to really like you. Can we start off with a condensed version of your story?

I’d like all your readers to know I’m just doing this for the lunch you promised me [laughs]. Let the record state I’m going to get free Wendy’s. And if you’re buying, I’m getting the large fries.

I was born in the early 1970s to baby boomer parents. My mother stayed at home and my father worked as a mechanic. There wasn’t enough money for me and my two younger sisters to go to college, so I moved to a larger town 20 minutes up the road to look for a job. Worked in local supermarket for about a year until I met the owner/operator of one of the local plumbing shops, who was there unclogging the drains in the meat department. I started asking questions about plumbing, and the next thing I knew he offered me a job as an apprentice.

So you’re what, 19 at this point?

Yeah, pretty much. He was in his 50s and smoked about a pack a minute. His kids weren’t interested in the business and it was really obvious he was looking for someone to take over. I registered in the apprenticeship program, met a girl around the same time, and celebrated becoming a journeyman by marrying her.

After another year, my boss tells me he’s ready to retire. He sells me the business for $5,000, which mostly consisted of tools and a van. He steps back into a part-time role, telling me to call him when I need the help. That didn’t really last long. Like I said, he was interested in retiring.

My wife had gotten a secretary job at the local high school right after we got married, so she was only able to help after supper with the billing. It was tough for the first couple of years, but we ended up figuring it out.

After I paid off the $5,000 loan, which was around the time my oldest was born, I started getting into investing. Having a child really kicked my ass, I was extra motivated to try and make a better life for her. I went down to the library and took out every book they had on investing and started getting my feet wet.

Can you remember any of the books you read?

The Wealthy Barber was the first one. It was great, it was all I needed on the personal finance side. I understood how to pay myself first almost immediately, but it didn’t really teach me how to invest. I probably read 20 investment books, but the only ones I remember were One Up on Wall Street and Beating the Street by Peter Lynch and Common Stocks and Uncommon Profits by Phil Fisher.

I also enjoyed The Millionaire Next Door and Buffett: The Making of an American Capitalist.

When did you actually start investing?

It was in 1996. I put the bulk of our savings into mutual funds at one bank, but I started working with a broker at another bank with about 20% of our money. I remember they were trying to get new investors in, so they offered a sale of $50 per trade. People have no idea how bad it used to be.

Anyway, we saved around $10,000 per year back then on a gross salary of around $60,000, so I really only had around $2,000 per year to invest in individual stocks. I spent a lot of time researching. I knew those commissions would kill my returns if I traded in and out, so I wanted by buy stocks that I could hold for a very long time.

My bank (CIBC) started offering online trades in 2000, so I moved my brokerage account there.

Do you remember the first stock you bought?

Yeah, I do. It was Seagram’s. It was one of the biggest blue chips and I really liked their products (laughs). They had a bunch of other assets as well, and I thought the sum of the parts was worth more than the share price. I sold a few years later when it merged with Vivendi.

The next year I bought Royal Bank shares. I still own those today, plus a few more.

How did you research stocks back then?

I used the internet, signing up for it as soon as someone offered it in our town. That was 1996 or so. We already had a computer because we were using it for the business. Our library also had copies of The Globe and Mail which I would read while my daughters would find books to take out. The Saturday edition was always the best.

One of the stock message boards I’d read gave me the idea to write companies I was interested in and ask them for their annual reports. I did this for years. I ended up with a stack of annual reports about three feet high. I would read them to try and learn about businesses, but I’m the first to admit I’d mostly skim them. I really didn’t have any idea what I was doing back then (laughs).

You mentioned you had the bulk of your money in mutual funds. When did you decide to go entirely independent?

It’s actually a really funny story. My advisor had a thing for gold and oil. She convinced me to put 20% of my capital into a gold fund and 20% into an oil fund and then the rest into a tech heavy equity fund. In hindsight the specialty funds had huge fees, but I didn’t care. I saw the rest of my stocks shooting higher and I wanted to put more money into the Nortels of the world. She pushed back, so after a few years I got mad and took my money out.

I was busy with the business, so I didn’t really have time to do any research. Plus, to be honest, I was overwhelmed. I didn’t really know what to do. So I stuck about 80% of my portfolio into a one-year GIC. This was around March, 2000, and the TSE was around 10,000. I go to cash out my GICs a year later and the TSE is around 7,000 and falling.

I’d like to take credit for calling the top of the tech bubble, but I really just got lucky.

How did you end up deciding on a more dividend oriented approach? 

My GIC was expiring and I knew I needed to figure out a plan.

I was doing a job for Terry, another local business owner. It turns out he’s an investor, so I start asking questions. His philosophy was to always buy blue chip stocks that paid dividends. He explained why and it made a lot of sense to me. I had also been paying attention to the big market sell-off and was noticing those stocks were down a whole lot less than the rest of the market.

I also remember many of the investing books I read telling me about the importance of dividends.

Probably the most important lesson Terry taught me was all about keeping it simple. He convinced me I didn’t need to be smarter than everybody else, I just needed to make sure I didn’t act stupid.

Based on that conversation I came up with a few basic ground rules:

1. I started, as you say, insisting on dividends. Because at least that way a bad decision would be minimized

2. I only bought what I deemed to be the highest quality stock in each sector

3. I sold my tech stocks (mostly Nortel) and swore off tech for a while. Tech didn’t pay a dividend and was too volatile for my new blue chip strategy.

4. I wanted stocks that would be relatively predictable and wouldn’t blow up in the next bear market

Can you share some of the stocks you bought for your portfolio back then?

Sure. I bought more Royal Bank, which I thought was the best of all the Canadian banks. I bought Enbridge because of its dependable cash flow. I bought Sobeys — which was eventually folded into Empire Company — because grocery was boring and predictable. I also bought Saputo after reading about its expansion plan, figuring milk and cheese made a pretty dull business.

The early 2000s were also when income trusts really started to gain popularity, so I bought Rogers Sugar, Boston Pizza, and Davis and Henderson. I saw all the double-digit yields and worried the payouts weren’t sustainable, so I really wanted to focus on quality.

There were some misses too. I bought Retirement Residences REIT. And I bought Penn West as my oil play. Both did okay because they had such big dividends, but they were both enough to persuade me to stick to higher quality, lower yielding names.

How many of these stocks do you still own?

I sold anything that cut the dividend (Boston Pizza, Penn West, and Retirement Residences REIT) and Davis and Henderson was taken private. Other than that, I own them all and would buy more when shares were depressed. For example, I doubled my position in Empire in 2016 when everyone hated the Safeway acquisition.

When we met for the first time you told me the 15 years between 2001 and 2016 “were basically just a blur.” Can you elaborate a little more on that and how you invested in that time? 

A few different things started happening at the same time. Firstly, I had two daughters [in the mid-to-late 1990s], and they kept me really busy. They had a full schedule of dance, sports, school events, etc. That part became better in the 2010s when they became teenagers, but before that it seemed like they were always doing something and I always had to drive them.

The business also picked up. Alberta’s economy really started booming in the 2000s, and one of my competitors retired right around the same time. Our town added a bunch of houses and got bigger, and I would install the plumbing in these new houses above and beyond my regular business.

The good part of being so busy is the business made a lot of money. I raised rates, stopped advertising, and basically just tried to maximize my income. We had a household income of six figures starting in about 2003 and it steadily went up until about 2016.

I didn’t have the time I wanted to research stocks, so I really just focused on doing simple things that made sense. I paid off the mortgage. I put money aside for my girls so they wouldn’t have to pay for college. I moved some money into USD when the CAD was worth more than the USD in 2010, and used that to buy Coca-Cola, Altria, Realty Income, and Walmart. I eventually reinvested those dividends into Apple shares, but that wasn’t until about 2016.

I picked those stocks because they met my quality/blue chip/boring/dividend paying criteria. I figured a rising dividend was probably one of the best signs of strength, so I started focusing more on dividend growth.

On the Canadian side, I bought Shaw, Reitmans, Investors Group, Manulife, Tim Hortons, RioCan, SmartCentres (back when it was Calloway), Boardwalk, Telus, Intact, and TC Energy, and I continued to buy stocks like Royal Bank, Fortis, and Enbridge that were working.

What I’d do is accumulate savings and then once a quarter I’d move money into my brokerage account. That new money was combined with dividends and invested. I’d alternate between my account and my wife’s account so we’d end up with around equal amounts in retirement.

I also immediately sold anything that cut the dividend. It’s a simple rule but it’s been a good one over the years. I did it with Reitmans, Manulife, RioCan, and Boardwalk when they cut dividends, and usually I’d cycle the proceeds into what I viewed was the highest quality name in the same sector. Manulife cut, so I bought Sun Life instead. Boardwalk cut, so I bought Northview Apartment REIT. And so on.

Things got a little less busy in 2016. The recession hit Alberta and my new build business dried up. I was back to a 40-hour work week again, my daughters were off in [large nearby city] going to university, and I finally had time to breathe. I spent a lot of time online reading about stocks, where I originally came across your [Nelson’s] writing at Motley Fool.

Can you share how much your portfolio was worth at that point and your dividend income?

It was right around $1.5M and dividends were $50k or so.

At that point you have no mortgage, your daughters are off to school, and you have very low household expenses. You were in the position to retire early. But you didn’t. Why?

I really started thinking about early retirement at that point, but I realized as a business owner that I could simply say no to more jobs and sort of create my own early retirement. I also raised my rates.

I started doing that, and within two weeks a competitor phoned me and said “hey, are you okay? Three people said you told them to call me.” He was very grateful because of how slow it was then, but those were my worst customers.

Mostly I didn’t pull the trigger because I thought I was going to be bored. I couldn’t envision what I’d do all day. So I went back to work — just not as hard as before.

Intermission

Please help support long-term financial writing and subscribe to Canadian Dividend Investing. Paid subscribers get:

• Two long-form posts per week analyzing specific Canadian dividend stocks

• Two model portfolios, one that aims to generate a 5% yield (with 4% dividend growth), and the other that aims to generate 8-10% annual dividend growth

• An ever-expanding library of one-page reports on various Canadian dividend stocks

• Much more!

Like Jim, I achieved financial independence and dividends were a big part of the reason why. If you’re looking to make sense of markets with a simple strategy anyone can execute, I can help.

Plus, new subscribers get 10% off! Regular price for a one-year subscription is $150, on sale for $135. Sale ends today, so upgrade your subscription today!

When did you finally decide to retire?

It was 2021. Alberta was in the middle of another recession because oil prices were low. The portfolio had bounced back from the 2020 lows. I was tired and sore all the time. I had been a plumber for 30-plus years and I was getting tired of unclogging drains and getting shit on my hands.

Also, Terry had retired in 2019, and he got to ride out the pandemic in Puerto Vallarta. This led to a bit of a midlife crisis, with me asking myself a lot of questions about why I was working so hard if I was already financially independent.

We talked about your little midlife crisis when we first met, which I thought was really interesting because I went through something similar. Can you talk more about your thought process in 2021 when you were about ready to retire?

Terry retiring to Mexico was a big thing. We had started investing around the same time, had similar portfolios, yet he was in Mexico six months of the year and I was stuck unclogging sinks. Frankly, I was jealous.

I also compared myself to my competitor, the one who was so grateful for my reject business in 2016. He spent every nickel he made and he had zero savings. I had invested every dollar I could for thirty years, yet we had pretty much identical lives. We still each got up each morning and went to work. He had to. But I didn’t.

I had also sacrificed a lot to get ahead. I worked 50+ hour weeks when the business was busy. I hated turning down work and letting people down. We didn’t take many holidays while my daughters were growing up. I really felt like I owed myself and my family this to make up for all my sacrifice.

Then, one day, I’m moving money into the brokerage account and realized the value of my portfolio had surpassed $3M. That was the piece of straw that broke the camel’s back, and I made the decision right then and there. I wrapped up my business right when the 2021 school year ended.

You said your portfolio was $3M when you retired. How much was your dividend income? And how does that compare to your dividend income today?

Between 2015 and 2021 I tended to buy stocks with a bit of a higher yield, so my portfolio yield increased from just over 3% to right around 4%. Dividend income was just shy of $120k when I retired, split pretty much equally between my wife and me.

My dividend income goes up 4-5% every year, so it’ll be around $125k in 2023 and probably around $130k in 2024.

Can you share your portfolio composition with everyone?

This is why I wanted to stay anonymous. I knew this question was coming (laughs).

Here it is, right down to the amount of income generated by each position.

This is great. I have many questions about your portfolio, and I’m sure the readers do too. I’ll start by asking how this portfolio is different than the one you retired with in 2021. 

The big difference was my position in Shaw. Shaw received the takeover offer from Rogers in spring of 2021, I believe, and my position shot up 50% overnight. That was the move that put me above $3M. I held until April this year, and got my full $40.50 from Rogers. I cycled that money into the more growthier stocks at the bottom of my portfolio — CNR, both Brookfield entities, TD Bank, and Metro. It temporarily cut my dividend income, but I needed more growth in my portfolio.

Do you worry about having 30% of your portfolio in just three names?

Not at all. These are great stocks that have been around a long time. All three are leaders in sectors Canadians need to survive. I’ve also held all three for about 25 years now, and I’m really comfortable with them and the risks they each have. I don’t lose a minute’s sleep over it.

Your portfolio value is down about $200k since you retired. Are you concerned with running out of money?

Nah. I’ve withdrawn $100k (total) from the portfolio for our living expenses for the last couple years. So that’s part of the reason it’s down. The other reason it’s down is most of my top holdings are lower than what they were in 2021. Enbridge is down 10%. SmartCentres is down 20%. Telus is down like 25%. TC Energy is down too.

I don’t spend a lot of time worrying about the stock values. I know they’ll come back. It’s my dividends I worry about.

So there’s no plans to change the strategy and sell out of some of those beaten up names?

Not at all. I’ve been through numerous market corrections before, and I’m not too worried about this one. These are good companies. They’ll figure it out.

What I have been doing is going though and thinking about clearing out a couple of stocks that aren’t increasing earnings per share. Extendicare and Automotive Properties REIT are candidates, but I don’t want to sell them at lows. So I’ll wait until they recover and bit and make a decision then.

How long have you waited for those two to work out?

I’ve owned Extendicare since 2014 and Automotive Properties since 2016.

That is a long time.  

Not really. I’ve owned many stocks over the years that have done nothing for years and then, boom, they do something. Patience is usually rewarded, but I know I’m never going to bat 1.000.

What I’ve found is when you hold boring dividend stocks even your misses aren’t bad. For example, I held IGM Financial for like a decade. Bought in 2012, finally got rid of it in 2022. The stock basically did nothing for ten years and I broke even on it, price wise.

But I collected a 6% dividend the whole time, and that dividend was reinvested in other stocks. That’s not a great result, but it could’ve easily been worse. Or I could’ve made 2% in a GIC.

Does this mean you don’t care about price appreciation?

Of course I want my stocks to go up. Like we’ve talked about, underlying earnings are what matters, and when I look at stocks I’m trying to see how earnings go higher. I want growth. Everyone should want growth.

But when I screw up and that growth doesn’t happen, then I at least know I’m protected by that dividend. Paying out those earnings to the owners also makes sure management doesn’t do anything stupid with the money — like most share buybacks.

Whoa whoa whoa whoa whoa. Are you saying you hate share buybacks? You’re going to get some people excited here, Jim.

Eh. Call me buyback agnostic. I think they’re a good concept that gets screwed up by a lot of management teams.

The problem is it’s easy to repurchase shares when things are going good. It’s much harder to do when things are tough. But that’s exactly when you should be repurchasing. Management’s first instinct is always to protect the balance sheet and conserve cash. No! Buy back the cheap stock, you cowards!

I’m like you — I think a company that repurchases 2% or 3% or whatever of the company every year, like clockwork, is much better than one that suspends their buyback the minute things look a little gloomy.

That was one of the reasons I bought Imperial Oil back when I did. It was buying back shares when nobody liked oil. But to be honest I mostly bought it for the balance sheet.

This is a good segue into your thoughts on the dividends versus no dividend debate. People can probably guess where you’re going to be on this issue, but you have what I thought was an interesting way of looking at it.

Okay, so I’m on Twitter, but I don’t say anything. I just follow people and read stuff. What’s that called? Loitering?

You’re a lurker.

Yeah, that’s it. And every time that debate comes up I’m so tempted to break my vow of silence. Because the total return side’s argument is academic bullshit that doesn’t happen in the real world.

I’ve been investing a long time and I’ve seen a lot of bad mergers and acquisitions. Most management teams aren’t very good at it. And we somehow expect them to be better at it with more money on the balance sheet? That money will burn a hole in their pocket and they’ll do dumb emotional shit with it.

So then they’re supposed to do share buybacks with it. Okay, but like I said earlier, most management teams are bad at buybacks. They won’t magically become better with more cash available to do buybacks with. They’ll get worse, guarantee it.

Okay, so the average investor is just supposed to sell their shares when they want cash? Dude, the average investor is TERRIBLE at selling. The first drawdown he’s going to panic and sell way too much in some misguided way of conserving value.

Academics don’t know what it’s like watching years of retirement savings evaporate during a crash. People do really dumb things when that happens. But they can’t help it.

Royal Bank’s dividend policy is best. They keep 50% of earnings to do whatever with, and I get 50% of earnings as a dividend. Perfect. What a great compromise.

How about the tax argument? The one that says dividends are a tax inefficient way to get your share of earnings?

Well I paid $0 in taxes in 2022, so I’d say dividends are pretty damn tax efficient (laughs).

I think most regular people do the majority of their investing inside RRSPs and TFSAs, and those are very tax efficient if you use them right.

Even when I was paying tax on my dividends because they were over and above my income, the taxes weren’t that bad. Dividends are taxed very well in this country.

Okay, we gotta end this soon because it’s getting long. So short answers for the last few questions . Are you worried about any of the stocks in your portfolio cutting their dividends?

I’m a little worried about SmartCentres, yeah. I still trust Mitch (Goldhar, CEO & Chairman) but that payout ratio is too high.

Thoughts about the recent Enbridge acquisition?

I like that they’re getting more into the utility sector. Price seems okay too. But they’re really exposed to higher rates. If rates keep heading higher they’re in trouble.

What are a couple of things that allowed you to achieve financial independence so far ahead of others?

I got lucky that plumbing ended up being so lucrative. I didn’t expect it to be. And we were fortunate to end up in rural Alberta. I paid $77k for my house in the mid-90s, a nice 1,200 square foot place with a full basement. Paid it off less than 10 years later. Not having a mortgage really helped, but even when we had one the payment was very reasonable.

I know you talk about this sometimes. Small towns are excellent for getting ahead. Housing is cheap and there’s nothing to go spend your money on. But I can understand why people like the city.

What do you do all day now that you’re retired?

I play a lot of pickleball. I really like pickleball. We visit friends and our kids. My oldest just bought a house and I replaced some of the plumbing for her. We grew a garden this year, which was fun. Terry invited me for an extended stay in Mexico this winter I’m very much looking forward to.

Will you come and answer questions when this gets published?

Sure, but I still don’t think my story is that interesting. I bet most of your readers are in the same spot.

Well, that ended up a lot longer than I first planned. But Jim is legit one of the more interesting guys I’ve met. His path to financial independence really spoke to me, maybe because it was much like my own.

I also really enjoyed seeing his portfolio, and I appreciate him sharing it. That’s not easy to do, even when you’re anonymous.

Feel free to ask Jim questions in the comments. And if you’re new here, please subscribe. We’ll do more of these types of interviews if y’all are interested.