Madison moved into the infamous mansion when she was 21 and soon became Hefner's 'number one girlfriend'.
Their relationship lasted seven years, and now the model is opening up about her time with the Playboy founder, who passed away in 2017 aged 91 - and let's just say it wasn't all parties and glamour.
Speaking on the Power: Hugh Hefner podcast, she revealed that the rules of the house were 'pretty restrictive' and said the only one she knew about before she moved in was the 9pm curfew.
She said: "Everybody would talk about that, the staff at the Mansion, the guests, everybody knew about it. People would kind of joke about it because it sounded so silly.
"I knew that was a rule before I moved in, but I didn't mind that rule for myself personally, because I was just at a phase in my life where I wasn't interested in going out partying on my own time, or trying to meet celebrities or trying to meet guys.
"I was interested in working hard and getting my life together. And keep in mind, when I decided to move in, I thought I would be living there for a year max because that was the example I had seen come before me."
With the early curfew, Holly didn't have a lot of time to go out dating - which was just as well because this was another thing that was banned by Hefner.
She continued: "You weren't allowed to date other guys, which I was fine with too, because I wasn't interested in dating at that point in my life.
"You weren't allowed to have an apartment of your own, which while I lived there, there were some women who did, they just kept that a secret from him."
The 41-year-old, who previously compared the Playboy Mansion to a cult, said that as well as attempting to control their love lives, Hefner set strict expectations about how the women should look.
"You were expected to look a certain way obviously, you had to have the blonde hair," she said, adding that there was pressure on what to wear.
She explained: "You were given $1000 (roughly £750) a week that was supposed to be a clothing allowance."
While this sounds like a good rule, the allowance was reportedly strictly managed - according to Holly, if Hefner's secretaries found out the money was spent on anything other than clothes then they would have the amount reduced.
She said: "So there was always this back and forth where he would want to not give the girls as much money, and they would have to come back and ask for it. It was weird.
"So there was this weird balancing act of trying to make it look like I had so much to show for that money, but save a little bit away so I could live if I decided to leave at a moment's notice."
In the same podcast, Holly - who left the mansion in 2008 - accused Hefner of taking unsolicited photos of playmates in order to control them.
"He's constantly taking all these pictures, and he would make copies of all the pictures and hand them out to everyone who had gone out that night," she said, comparing the act to 'revenge porn'.
She explained: "I don't know if he just assumed that was OK because all these women want to be in the magazine so bad - they must be OK with getting naked, so I'm going to take pictures while they're wasted and just hand those pictures out.
"That's the kind of thing that can make you feel kind of stuck in a situation or overinvested. It's one of those things that makes you feel a little more backed into a corner."
Words: Daisy Phillipson