No, there is no point on Earth where all directions can be considered "down." "Down" is a relative term that refers to the direction toward the center of the Earth due to gravity. Since the Earth is roughly spherical in shape, the force of gravity pulls objects toward its center. Therefore, standing at any location on the Earth's surface, the direction of "down" is always perpendicular to the surface, pointing toward the Earth's core.
While there are regions such as the summit of a mountain or the bottom of a deep ocean trench where the local gravitational field may vary slightly from the average, "down" would still be directed toward the center of the Earth in those locations. In other words, the concept of "down" is always determined by the local gravitational field and the direction toward the Earth's center.