There are many different names for popping up a "window" using javascript. There are also many different extensions with different styles and features to help you do this:

JED - Popups & iFrames

The basic modal window functionality is built into Joomla using MooTools and can be used anywhere on your site. To do this, you need to tell Joomla to include MooTools using the following PHP call:

Joomla API - JHTML Behavior: MooTools

JHTML::_('behavior.mootools');

This will tell Joomla to include the following javascript file:

<script type="text/javascript" src="/media/system/js/mootools.js"></script>

The PHP call to tell Joomla to include modal functionality looks similar to the mootools call above:

Joomla API - JHTML Behavior: Modal

JHTML::_('behavior.modal');

This will tell Joomla to include the following javascript file, script block, and stylesheet:

<script type="text/javascript" src="/media/system/js/modal.js"></script>
<script type="text/javascript">
  window.addEvent('domready', function() {<
    SqueezeBox.initialize({});
    $$('a.modal').each(function(el) {
      el.addEvent('click', function(e) {
        new Event(e).stop();
        SqueezeBox.fromElement(el);
      });
    });
  });
</script>
<link rel="stylesheet" href="/media/system/css/modal.css" type="text/css" />

Once these are all included, you can use the Joomla SqueezeBox functionality to make your links open up in javascript modal windows rather than opening in a new window. The SqueezeBox offers different handler types and options. The following are a few of the commonly used cases. For a full understanding of the SqueezeBox, see the modal.js file included in Joomla.

Handler: Adopt

Handler: iFrame

iFrame

Handler: Image

Handler: String

String - This one doesn't work. I believe there is a bug in modal.js.

Handler: Url

Url